Skip to content

Make Extensions linker friendly. - #40431

Merged
eerhardt merged 5 commits into
dotnet:masterfrom
eerhardt:AnnotateExtensions
Aug 7, 2020
Merged

Make Extensions linker friendly.#40431
eerhardt merged 5 commits into
dotnet:masterfrom
eerhardt:AnnotateExtensions

Conversation

@eerhardt

@eerhardt eerhardt commented Aug 5, 2020

Copy link
Copy Markdown
Member

Annotating the rest of the Microsoft.Extensions libraries.

Fix #40396

With these changes, there are the following linker warnings left in Microsoft.Extensions assemblies:

https://github.com/mono/linker/issues/1412
F:\git\runtime2\src\libraries\Microsoft.Extensions.Configuration.Abstractions\src\ConfigurationExtensions.cs(22,9): Trim analysis warning IL2006: Microsoft.Extensions.Configuration.ConfigurationExtensions.Add<TSource>(IConfigurationBuilder,Action<TSource>): The generic parameter 'TSource' from 'Microsoft.Extensions.Configuration.ConfigurationExtensions.Add<TSource>(IConfigurationBuilder,Action<TSource>)' with dynamically accessed member kinds 'None' is passed into the generic parameter 'T' from 'System.Activator.CreateInstance<T>()' which requires dynamically accessed member kinds 'PublicParameterlessConstructor'. To fix this add DynamicallyAccessedMembersAttribute to it and specify at least these member kinds 'PublicParameterlessConstructor'.

https://github.com/mono/linker/issues/1416
F:\git\runtime2\src\libraries\Microsoft.Extensions.Http\src\DependencyInjection\HttpClientBuilderExtensions.cs(339,13): Trim analysis warning IL2006: Microsoft.Extensions.DependencyInjection.HttpClientBuilderExtensions.<>c__DisplayClass10_0<TClient>.<AddTypedClientCore>b__0(IServiceProvider): The generic parameter 'TClient' from 'Microsoft.Extensions.DependencyInjection.HttpClientBuilderExtensions.<>c__DisplayClass10_0<TClient>' with dynamically accessed member kinds 'None' is passed into the generic parameter 'TClient' from 'Microsoft.Extensions.Http.ITypedHttpClientFactory<TClient>' which requires dynamically accessed member kinds 'PublicConstructors'. To fix this add DynamicallyAccessedMembersAttribute to it and specify at least these member kinds 'PublicConstructors'.
F:\git\runtime2\src\libraries\Microsoft.Extensions.Http\src\DependencyInjection\HttpClientBuilderExtensions.cs(339,13): Trim analysis warning IL2006: Microsoft.Extensions.DependencyInjection.HttpClientBuilderExtensions.<>c__DisplayClass10_0<TClient>.<AddTypedClientCore>b__0(IServiceProvider): The generic parameter 'TClient' from 'Microsoft.Extensions.DependencyInjection.HttpClientBuilderExtensions.<>c__DisplayClass10_0<TClient>' with dynamically accessed member kinds 'None' is passed into the generic parameter 'TClient' from 'Microsoft.Extensions.Http.ITypedHttpClientFactory<TClient>' which requires dynamically accessed member kinds 'PublicConstructors'. To fix this add DynamicallyAccessedMembersAttribute to it and specify at least these member kinds 'PublicConstructors'.
F:\git\runtime2\src\libraries\Microsoft.Extensions.Http\src\DependencyInjection\HttpClientBuilderExtensions.cs(339,13): Trim analysis warning IL2006: Microsoft.Extensions.DependencyInjection.HttpClientBuilderExtensions.<>c__DisplayClass10_0<TClient>.<AddTypedClientCore>b__0(IServiceProvider): The generic parameter 'TClient' from 'Microsoft.Extensions.DependencyInjection.HttpClientBuilderExtensions.<>c__DisplayClass10_0<TClient>' with dynamically accessed member kinds 'None' is passed into the generic parameter 'TClient' from 'Microsoft.Extensions.Http.ITypedHttpClientFactory<TClient>' which requires dynamically accessed member kinds 'PublicConstructors'. To fix this add DynamicallyAccessedMembersAttribute to it and specify at least these member kinds 'PublicConstructors'.
F:\git\runtime2\src\libraries\Microsoft.Extensions.Http\src\DependencyInjection\HttpClientBuilderExtensions.cs(402,13): Trim analysis warning IL2006: Microsoft.Extensions.DependencyInjection.HttpClientBuilderExtensions.<>c__DisplayClass12_0<TClient,TImplementation>.<AddTypedClientCore>b__0(IServiceProvider): The generic parameter 'TImplementation' from 'Microsoft.Extensions.DependencyInjection.HttpClientBuilderExtensions.<>c__DisplayClass12_0<TClient,TImplementation>' with dynamically accessed member kinds 'None' is passed into the generic parameter 'TClient' from 'Microsoft.Extensions.Http.ITypedHttpClientFactory<TClient>' which requires dynamically accessed member kinds 'PublicConstructors'. To fix this add DynamicallyAccessedMembersAttribute to it and specify at least these member kinds 'PublicConstructors'.
F:\git\runtime2\src\libraries\Microsoft.Extensions.Http\src\DependencyInjection\HttpClientBuilderExtensions.cs(402,13): Trim analysis warning IL2006: Microsoft.Extensions.DependencyInjection.HttpClientBuilderExtensions.<>c__DisplayClass12_0<TClient,TImplementation>.<AddTypedClientCore>b__0(IServiceProvider): The generic parameter 'TImplementation' from 'Microsoft.Extensions.DependencyInjection.HttpClientBuilderExtensions.<>c__DisplayClass12_0<TClient,TImplementation>' with dynamically accessed member kinds 'None' is passed into the generic parameter 'TClient' from 'Microsoft.Extensions.Http.ITypedHttpClientFactory<TClient>' which requires dynamically accessed member kinds 'PublicConstructors'. To fix this add DynamicallyAccessedMembersAttribute to it and specify at least these member kinds 'PublicConstructors'.
F:\git\runtime2\src\libraries\Microsoft.Extensions.Http\src\DependencyInjection\HttpClientBuilderExtensions.cs(402,13): Trim analysis warning IL2006: Microsoft.Extensions.DependencyInjection.HttpClientBuilderExtensions.<>c__DisplayClass12_0<TClient,TImplementation>.<AddTypedClientCore>b__0(IServiceProvider): The generic parameter 'TImplementation' from 'Microsoft.Extensions.DependencyInjection.HttpClientBuilderExtensions.<>c__DisplayClass12_0<TClient,TImplementation>' with dynamically accessed member kinds 'None' is passed into the generic parameter 'TClient' from 'Microsoft.Extensions.Http.ITypedHttpClientFactory<TClient>' which requires dynamically accessed member kinds 'PublicConstructors'. To fix this add DynamicallyAccessedMembersAttribute to it and specify at least these member kinds 'PublicConstructors'.

ConfigurationBinder recursively inspects properties - this can't be modeled with the linker - https://github.com/mono/linker/issues/1087
F:\git\runtime2\src\libraries\Microsoft.Extensions.Configuration.Binder\src\ConfigurationBinder.cs(371,17): Trim analysis warning IL2006: Microsoft.Extensions.Configuration.ConfigurationBinder.CreateInstance(Type): The parameter 'type' of method 'Microsoft.Extensions.Configuration.ConfigurationBinder.CreateInstance(Type)' with dynamically accessed member kinds 'None' is passed into the implicit 'this' parameter of method 'System.Reflection.TypeInfo.get_DeclaredConstructors()' which requires dynamically accessed member kinds 'NonPublicConstructors | PublicConstructors'. To fix this add DynamicallyAccessedMembersAttribute to it and specify at least these member kinds 'NonPublicConstructors | PublicConstructors'.
F:\git\runtime2\src\libraries\Microsoft.Extensions.Configuration.Binder\src\ConfigurationBinder.cs(380,17): Trim analysis warning IL2006: Microsoft.Extensions.Configuration.ConfigurationBinder.CreateInstance(Type): The parameter 'type' of method 'Microsoft.Extensions.Configuration.ConfigurationBinder.CreateInstance(Type)' with dynamically accessed member kinds 'None' is passed into the parameter #0 of method 'System.Object System.Activator::CreateInstance(System.Type)' which requires dynamically accessed member kinds 'PublicParameterlessConstructor'. To fix this add DynamicallyAccessedMembersAttribute to it and specify at least these member kinds 'PublicParameterlessConstructor'.
F:\git\runtime2\src\libraries\Microsoft.Extensions.Configuration.Binder\src\ConfigurationBinder.cs(403,13): Trim analysis warning IL2006: Microsoft.Extensions.Configuration.ConfigurationBinder.BindDictionary(Object,Type,IConfiguration,BinderOptions): The parameter 'dictionaryType' of method 'Microsoft.Extensions.Configuration.ConfigurationBinder.BindDictionary(Object,Type,IConfiguration,BinderOptions)' with dynamically accessed member kinds 'None' is passed into the implicit 'this' parameter of method 'System.Reflection.TypeInfo.GetDeclaredProperty(String)' which requires dynamically accessed member kinds 'NonPublicProperties | PublicProperties'. To fix this add DynamicallyAccessedMembersAttribute to it and specify at least these member kinds 'NonPublicProperties | PublicProperties'.
F:\git\runtime2\src\libraries\Microsoft.Extensions.Configuration.Binder\src\ConfigurationBinder.cs(433,13): Trim analysis warning IL2006: Microsoft.Extensions.Configuration.ConfigurationBinder.BindCollection(Object,Type,IConfiguration,BinderOptions): The parameter 'collectionType' of method 'Microsoft.Extensions.Configuration.ConfigurationBinder.BindCollection(Object,Type,IConfiguration,BinderOptions)' with dynamically accessed member kinds 'None' is passed into the implicit 'this' parameter of method 'System.Reflection.TypeInfo.GetDeclaredMethod(String)' which requires dynamically accessed member kinds 'NonPublicMethods | PublicMethods'. To fix this add DynamicallyAccessedMembersAttribute to it and specify at least these member kinds 'NonPublicMethods | PublicMethods'.
F:\git\runtime2\src\libraries\Microsoft.Extensions.Configuration.Binder\src\ConfigurationBinder.cs(565,17): Trim analysis warning IL2006: Microsoft.Extensions.Configuration.ConfigurationBinder.GetAllProperties(TypeInfo): The parameter 'type' of method 'Microsoft.Extensions.Configuration.ConfigurationBinder.GetAllProperties(TypeInfo)' with dynamically accessed member kinds 'None' is passed into the implicit 'this' parameter of method 'System.Reflection.TypeInfo.get_DeclaredProperties()' which requires dynamically accessed member kinds 'NonPublicProperties | PublicProperties'. To fix this add DynamicallyAccessedMembersAttribute to it and specify at least these member kinds 'NonPublicProperties | PublicProperties'.

The following are all 'MakeGenericMethod' - I'm not sure what we can do about these
F:\git\runtime2\src\libraries\Microsoft.Extensions.DependencyInjection\src\ServiceLookup\CallSiteFactory.cs(241,17): Trim analysis warning IL2006: Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteFactory.TryCreateOpenGeneric(ServiceDescriptor,Type,CallSiteChain,Int32): Calling to 'System.Type.MakeGenericType' on unrecognized value.
F:\git\runtime2\src\libraries\Microsoft.Extensions.DependencyInjection\src\ServiceLookup\Expressions\ExpressionResolverBuilder.cs(135,17): Trim analysis warning IL2006: Microsoft.Extensions.DependencyInjection.ServiceLookup.ExpressionResolverBuilder.VisitIEnumerable(IEnumerableCallSite,Object): Call to 'System.Reflection.MethodInfo.MakeGenericMethod' is not recognized.
F:\git\runtime2\src\libraries\Microsoft.Extensions.DependencyInjection\src\ServiceLookup\ILEmit\ILEmitResolverBuilder.cs(216,17): Trim analysis warning IL2006: Microsoft.Extensions.DependencyInjection.ServiceLookup.ILEmitResolverBuilder.VisitIEnumerable(IEnumerableCallSite,ILEmitResolverBuilderContext): Call to 'System.Reflection.MethodInfo.MakeGenericMethod' is not recognized.

@davidfowl

Copy link
Copy Markdown
Member

The following are all 'MakeGenericMethod' - I'm not sure what we can do about these

I think we have to suppress these. There's nothing we can flow from here. Any open generic registered into the container needs to do the needful and mark things appropriately.

@davidfowl

Copy link
Copy Markdown
Member

ConfigurationBinder recursively inspects properties - this can't be modeled with the linker - dotnet/linker#1087

This is unfortunate but understandable (especially because it's polymorphic)

@MichalStrehovsky

Copy link
Copy Markdown
Member

For the warnings that refer to the DisplayClass - we could work around by adding our own closure class that is properly annotated. Linker can't do much without trying to reverse engineer C# compiler operations and that's a sketchy territory (I don't think the Roslyn team would be happy if we were to take a dependency on their manglings).

For MakeGenericMethod - the risk is basically that if we can end up with MakeGenericMethod creating generic methods that have generic parameters annotated with dataflow annotations, linker no longer enforces the annotations (because linker doesn't run at runtime) and things could start failing. If we're convinced that can't happen, suppression is okay.

Basically suppression always means that things could start failing at runtime because we're turning off automatic checking. It needs to be done with extreme care because one missed spot and everything could fall apart with a hard to diagnose exception at runtime. We also need to remember that a method got MakeGenericMethod'd somewhere and we should not add dataflow annotations to it in the future because they won't be enforced.

@davidfowl

Copy link
Copy Markdown
Member

For the warnings that refer to the DisplayClass - we could work around by adding our own closure class that is properly annotated. Linker can't do much without trying to reverse engineer C# compiler operations and that's a sketchy territory (I don't think the Roslyn team would be happy if we were to take a dependency on their manglings).

Why can't we do something automagical for compiler generated types (like inherit attributes when they are from compiler generated types).

For MakeGenericMethod - the risk is basically that if we can end up with MakeGenericMethod creating generic methods that have generic parameters annotated with dataflow annotations, linker no longer enforces the annotations (because linker doesn't run at runtime) and things could start failing. If we're convinced that can't happen, suppression is okay.

Right, the way this system works it make sense to suppress. It can break but we'll need to document this pattern in general (open generics registered in DI have this problem and there's no single fix).

@MichalStrehovsky

Copy link
Copy Markdown
Member

Why can't we do something automagical for compiler generated types (like inherit attributes when they are from compiler generated types).

It requires threading the annotations through to the closure constructor, and to the instance fields. It's not impossible but it requires careful implementation of the right constructs that the C# compiler can emit for these.

We've been avoiding doing crossmethod dataflow analysis because it's slow to compute and starts to be complicated to both compute and report warnings on once things like virtual and interface methods are supported. Closure classes might be less complicated, but it still requires hardcoding C# compiler knowledge I would rather not hardcode.

@davidfowl

Copy link
Copy Markdown
Member

It needs to be solved though, if we change people's coding patterns because they can't write annotations on closures, it basically means you'll get warnings that need to be ignored forever. Forcing me to rewrite my code to add an annotation is also not great.

@MichalStrehovsky

Copy link
Copy Markdown
Member

The code needs to be written in a way that can be statically analyzed. This pull request is doing it in e.g. ProviderAliasUtilities.cs already.

There will be reflection code that will work as-is. There will be code where you add annotations and it will work. There will be code that you need to tweak before you can annotate it. There will be code that you need to throw out and write a source generator.

C# has a lot of these closure-based constructs where it will generate a lot of goo that can only be analyzed with whole program dataflow for something that is a one-liner in the C# source. There's yield return state machines. There's async state machines. There's lambda closures. There's the dynamic keyword. It's a lot of fragile concepts to hardcode into linker and C# is not done adding more.

@MichalStrehovsky

Copy link
Copy Markdown
Member

ConfigurationBinder recursively inspects properties

What's the failure mode when the properties are missing?

I have bad memories of missing properties in XAML databinding in .NET Native - XAML is pretty quiet when a property doesn't exist so a bug where we trimmed something that shouldn't have been trimmed usually manifested itself with "the UI looks off, but there's no exception thrown anywhere". Those were fun to debug.

@davidfowl

Copy link
Copy Markdown
Member

C# has a lot of these closure-based constructs where it will generate a lot of goo that can only be analyzed with whole program dataflow for something that is a one-liner in the C# source. There's yield return state machines. There's async state machines. There's lambda closures. There's the dynamic keyword. It's a lot of fragile concepts to hardcode into linker and C# is not done adding more.

Asking people rewrite their code is a non starter. It would be better if there were more well defined contracts defined between the linker and the compiler (that's why I initially focused on CompilerGeneratedAttribute).

@eerhardt
eerhardt requested a review from layomia August 7, 2020 14:59
@eerhardt
eerhardt merged commit 254ef0f into dotnet:master Aug 7, 2020
@eerhardt
eerhardt deleted the AnnotateExtensions branch August 7, 2020 19:36
@eerhardt

eerhardt commented Aug 7, 2020

Copy link
Copy Markdown
Member Author

ConfigurationBinder recursively inspects properties

What's the failure mode when the properties are missing?

From what I can tell it silently skips the properties without setters. So you can start getting NullReferenceException if you were expecting the properties to be set to valid objects after binding.

I've opened #40551 to make the Binder linker-safe.

It would be better if there were more well defined contracts defined between the linker and the compiler

Would it be possible for the compiler to flow the generic type attributes to the generated code? That doesn't seem unreasonable to me, and I think it could solve the problem with HttpClientBuilderExtensions.

@jkoritzinsky

Copy link
Copy Markdown
Member

Would it be possible for the compiler to flow the generic type attributes to the generated code? That doesn't seem unreasonable to me, and I think it could solve the problem with HttpClientBuilderExtensions.

That should be feasible and seems like a clean solution. Let's open an issue with the Roslyn team about it.

@eerhardt

eerhardt commented Aug 7, 2020

Copy link
Copy Markdown
Member Author

Let's open an issue with the Roslyn team about it.

I've opened dotnet/roslyn#46646 for this.

@vitek-karas

Copy link
Copy Markdown
Member

@eerhardt Why didn't we also update the ref assemblies with the new attributes?

Jacksondr5 pushed a commit to Jacksondr5/runtime that referenced this pull request Aug 10, 2020
* Make Extensions linker friendly.

Annotating the rest of the Microsoft.Extensions libraries.

Fix dotnet#40396

* Disable parallelism in trimming tests to fix race condition.
@eerhardt

Copy link
Copy Markdown
Member Author

Why didn't we also update the ref assemblies with the new attributes?

Mostly because I forgot to. I've opened #40621 which is adding them.

As an aside - these assemblies don't target net5.0, so the attributes in the ref assembly are all internal. Also, we have an infrastructure bug that isn't catching these problems on attributes on parameters and on generic types. cc @safern

@karelz karelz modified the milestones: 6.0.0, 5.0.0 Aug 18, 2020
@ghost ghost locked as resolved and limited conversation to collaborators Dec 7, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Make Microsoft.Extensions.Hosting linker friendly

8 participants